Search engine agents, often referred to as spiders or crawlers, navigate websites in a methodical manner and retrieve information about sites visited. For example, a crawler can make a copy of all or a portion of websites and related information. The search engine then analyzes the content captured by one or more crawlers to determine how a page will be indexed. Some engines will index all words on a website while others may only index terms associated with particular tags such as such for example: title, header or metatag(s). Crawlers must also periodically revisit webpages to detect and capture changes thereto since the last indexing.
Once indexes are generated, they typically are assigned a ranking with respect to certain keywords, and stored in a database. A proprietary algorithm is often employed to evaluate the index for relevancy, for example, based on frequency and location of words on a webpage, among other things. A distinctive factor in performance amongst conventional search engines is the ranking algorithm respectively employed.
Upon entry of one or more keywords as a search query, the search engine retrieves indexed information that matches the query from the database, generates a snippet of text associated with each of the matching sites and displays the results to a user. The user can thereafter scroll through a plurality of returned sites in connection with determining if the sites are related to interests of the user. However, this can be an extremely time-consuming and frustrating process as search engines often return a substantial number of sites. More often then not, the user is forced to further narrow the search iteratively by altering and/or adding keywords and Boolean operators to converge on websites that provide the sought after information.